Virtual interest segmentation

ABSTRACT

Systems and methods for virtual interest segmentation may include (1) performing a semantic segmentation of an image of a user&#39;s environment, captured by an artificial reality (AR) device being worn by the user, to identify objects within the user&#39;s environment, (2) in addition to performing the semantic segmentation, performing an interest segmentation of the image to determine a personal interest that the user may have in a particular object identified via the semantic segmentation, (3) creating virtual content relating to the particular object based on the user&#39;s personal interest in the particular object, and (4) displaying the virtual content within a display element of the AR device. Various other methods, systems, and computer-readable media are also disclosed.

BACKGROUND

Virtual and augmented reality systems (collectively known as artificialreality systems) are often configured to be worn by a user as the userinteracts with the real world. Such artificial reality systems typicallyinclude a display element through which the user may see the real world.The display element may additionally be configured to display virtualcontent such that the virtual content is visually superimposed over thereal world within the display element. The variety of content that maybe displayed is as vast as the variety of scenes and objects that may beviewed through the display element. As such, it may be difficult tonarrow this variety to content that is pertinent to a particular user.If the content is well-tailored to a user, the content can be a powerfuladdition to the user's interactions with the real world. However, if thecontent is not pertinent to the user, the content may be an irritatingdistraction. The present disclosure therefore identifies a need forsystems and methods for digitally generating and displaying virtualcontent to a user wearing an artificial reality system that ispersonally pertinent to the user.

SUMMARY

As will be described in greater detail below, the present disclosuredescribes various systems and methods for (1) digitally determiningwhich objects are in an environment of a user wearing an artificialreality (AR) device, and then (2) digitally determining a personalsignificance that one or more of those objects may have for the user(e.g., by performing a virtual interest segmentation). In one example, acomputer-implemented method for performing this task may include (1)performing a semantic segmentation of an image of a user's environment,captured by an artificial reality (AR) device being worn by the user, toidentify objects within the user's environment, (2) in addition toperforming the semantic segmentation, performing an interestsegmentation of the image to determine a personal interest that the usermay have in a particular object identified via the semanticsegmentation, (3) creating virtual content relating to the particularobject based on the user's personal interest in the particular object,and (4) displaying the virtual content within a display element of theAR device (e.g., such that the virtual content is superimposed over theparticular object within the display element).

In one example, determining the personal interest may include (1)identifying a type of object corresponding to the particular object and(2) determining that the user is interested in the type of object. Inthis example, the virtual content may include content configured to drawthe user's attention to the particular object.

In another example, determining the personal interest may includedetermining that (1) the particular object belongs to the user and/or(2) the user has interacted with the particular object more than athreshold amount. In this example, the virtual content may be createdand displayed in response to determining that the user has lost theparticular object and the virtual content may be configured to indicatea location of the particular object within the environment. In onespecific embodiment, the semantic and interest segmentations may beperformed at a first moment in time, during which the user wearing theAR device is within the environment while wearing the AR device. In thisembodiment, the user may be determined to have lost the particularobject at a second moment in time during which the user is no longerwithin the environment.

In some examples, the interest segmentation may be based on the user'shistorical eye-tracking data and/or the historical eye-tracking data ofone or more additional users with a particular similarity or bundle ofsimilarities to the user. In one embodiment, the interest segmentationmay be based on a heatmap of the environment that includes the user'seye-tracking data with respect to the environment. Additionally oralternatively, the interest segmentation may be based on GPS dataassociated with the user, URL browsing data associated with the user,and/or user-submitted data submitted by the user. In some examples, theinterest segmentation may be based on an interaction of the userrecorded by an additional device within an Internet of Things (IoT)system that includes the AR device.

In one example, the interest segmentation may be based on datadescribing ephemeral factors predicted to be affecting a current stateof the user (e.g., a time of day, an emotional state of the user, aphysiological state of the user, and/or a current activity of the user).In some examples, the interest segmentation may be performed by a neuralnetwork. In some embodiments, the interest segmentation may be performedand/or the virtual content may be displayed in response to receivinguser permission to do so.

In one embodiment, a system for implementing the above-described methodmay include (1) a segmentation module, stored in memory, that (i)performs a semantic segmentation of an image of a user's environment,captured by an AR device being worn by the user, to identify objectswithin the user's environment and (ii) performs an interest segmentationof the image to determine a personal interest that the user may have ina particular object identified via the semantic segmentation, (2) acontent creation module that creates virtual content relating to theparticular object based on the user's personal interest in theparticular object, (3) a display module that displays the virtualcontent within a display element of the AR device, and (4) a physicalprocessor configured to execute the segmentation module, the contentcreation module, and the display module.

In some examples, the above-described method may be encoded ascomputer-readable instructions on a non-transitory computer-readablemedium. For example, a computer-readable medium may include one or morecomputer-readable instructions that, when executed by at least oneprocessor of a computing device, may cause the computing device to (1)perform a semantic segmentation of an image of a user's environment,captured by an AR device being worn by the user, to identify objectswithin the user's environment, (2) perform an interest segmentation ofthe image to determine a personal interest that the user may have in aparticular object identified via the semantic segmentation, (3) createvirtual content relating to the particular object based on the user'spersonal interest in the particular object, and (4) display the virtualcontent within a display element of the AR device.

Features from any of the above-mentioned embodiments may be used incombination with one another in accordance with the general principlesdescribed herein. These and other embodiments, features, and advantageswill be more fully understood upon reading the following detaileddescription in conjunction with the accompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate a number of exemplary embodimentsand are a part of the specification. Together with the followingdescription, these drawings demonstrate and explain various principlesof the present disclosure.

FIG. 1 is an illustration of an exemplary AR system.

FIG. 2 is an illustration of an additional exemplary AR system.

FIG. 3 is an illustration of an exemplary virtual reality (VR) system.

FIG. 4 is an illustration of a flow diagram of an exemplary method forperforming an interest segmentation to generate virtual content for anAR device.

FIG. 5 is a block diagram of an exemplary system for performing aninterest segmentation to generate virtual content for an AR device.

FIG. 6 is an illustration of an exemplary real-world scene that may becaptured by an AR device and which may be the subject of an interestsegmentation.

Throughout the drawings, identical reference characters and descriptionsindicate similar, but not necessarily identical, elements. While theexemplary embodiments described herein are susceptible to variousmodifications and alternative forms, specific embodiments have beenshown by way of example in the drawings and will be described in detailherein. However, the exemplary embodiments described herein are notintended to be limited to the particular forms disclosed. Rather, thepresent disclosure covers all modifications, equivalents, andalternatives falling within the scope of the appended claims.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The present disclosure is generally directed to (1) digitallydetermining which objects are in an environment of a user wearing anartificial reality (AR) device, and then (2) digitally determining apersonal significance that one or more of those objects may have for theuser (e.g., by performing a virtual interest segmentation). A processfor performing interest segmentation may include (1) performing asemantic segmentation of an image of a scene (captured by a user's ARdevice) to identify and label objects in the scene and then (2)digitally analyzing the identified objects to identify a personalsignificance that one or more of the objects may have to the user. Anobject may be identified as personally significant to the user for avariety of reasons. Objects of significance might include objects thatbelong to the user (e.g., the user's phone or keys), objects that areoften frequented or used by the user (e.g., a particular park bench),and/or objects that are predicted to be of interest to the user (e.g.,electrical outlets in an airport, a certain type of restaurant, or acertain type of musical instrument). After performing the interestsegmentation, virtual content relating to the personally significantobjects may be created and displayed via the AR device, based on theresults of the interest segmentation.

A degree to which the interest segmentation is personalized may varybased on a privacy setting selected by the user. For example, theinterest segmentation may be performed by a neural network and the datacollected to be used as inputs to the neural network may be limited totypes of data that the user has given permission to have collected. Suchfactors might include (without limitation) the user's historicalgaze-tracking and/or hand-tracking data, user preferences submitted bythe user, the user's browsing history, contextual factors such as a timeof day, and/or a deduced emotional state of the user.

The virtual content may take many different forms and may include avariety of different information. For example, the virtual content maytake the form of (1) an alert pointing out an object in an environmentof the user that has been predicted to be of interest to the user and/or(2) additional information about an object in the environment that hasbeen predicted to be of interest to the user (e.g., historicalinformation about a building, a cost of a product, an establishment'shours of operation, etc.).

In some embodiments, the interest segmentation may be based at leastpartially on aggregated historical data collected from other users of ARdevices. Users may provide permission for their data to be added to thisaggregated data ex ante or post hoc.

As will be explained in greater detail below, embodiments of the presentdisclosure may improve an AR device's ability to meaningfully presentdigital content to a user. This may improve the functioning of acomputer itself (i.e., an AR device) by increasing the computer'sutility.

Embodiments of the instant disclosure may include or be implemented inconjunction with various types of artificial reality systems. Artificialreality is a form of reality that has been adjusted in some mannerbefore presentation to a user, which may include, e.g., a virtualreality (VR), an augmented reality (AR), a mixed reality (MR), a hybridreality, or some combination and/or derivative thereof.

Artificial reality content may include completely generated content orgenerated content combined with captured (e.g., real-world) content. Theartificial reality content may include video, audio, haptic feedback, orsome combination thereof, any of which may be presented in a singlechannel or in multiple channels (such as stereo video that produces athree-dimensional effect to the viewer). Additionally, in someembodiments, artificial reality may also be associated withapplications, products, accessories, services, or some combinationthereof, that are used to, e.g., create content in an artificial realityand/or are otherwise used in (e.g., to perform activities in) anartificial reality.

Artificial reality systems may be implemented in a variety of differentform factors and configurations. Some artificial reality systems may bedesigned to work without near-eye displays, an example of which is ARsystem 100 in FIG. 1. Other artificial reality systems may include anear-eye display that also provides visibility into the real world(e.g., AR system 200 in FIG. 2) or that visually immerses a user in anartificial reality (e.g., VR system 300 in FIG. 3). While someartificial reality devices may be self-contained systems, otherartificial reality devices may communicate and/or coordinate withexternal devices to provide an artificial reality experience to a user.Examples of such external devices include handheld controllers, mobiledevices, desktop computers, devices worn by a user, devices worn by oneor more other users, and/or any other suitable external system.

Turning to FIG. 1, AR system 100 generally represents a wearable devicedimensioned to fit about a body part (e.g., a head) of a user. As shownin FIG. 1, system 100 may include a frame 102 and a camera assembly 104that is coupled to frame 102 and configured to gather information abouta local environment by observing the local environment. AR system 100may also include one or more audio devices, such as output audiotransducers 108(A) and 108(B) and input audio transducers 110. Outputaudio transducers 108(A) and 108(B) may provide audio feedback and/orcontent to a user, and input audio transducers 110 may capture audio ina user's environment.

As shown, AR system 100 may not necessarily include a near-eye displaypositioned in front of a user's eyes. AR systems without near-eyedisplays may take a variety of forms, such as head bands, hats, hairbands, belts, watches, wrist bands, ankle bands, rings, neckbands,necklaces, chest bands, eyewear frames, and/or any other suitable typeor form of apparatus. While AR system 100 may not include a near-eyedisplay, AR system 100 may include other types of screens or visualfeedback devices (e.g., a display screen integrated into a side of frame102).

The embodiments discussed in this disclosure may also be implemented inAR systems that include one or more near-eye displays. For example, asshown in FIG. 2, AR system 200 may include an eyewear device 202 with aframe 210 configured to hold a left display device 215(A) and a rightdisplay device 215(B) in front of a user's eyes. Display devices 215(A)and 215(B) may act together or independently to present an image orseries of images to a user. While AR system 200 includes two displays,embodiments of this disclosure may be implemented in AR systems with asingle near-eye display or more than two near-eye displays.

In some embodiments, AR system 200 may include one or more sensors, suchas sensor 240. Sensor 240 may generate measurement signals in responseto motion of AR system 200 and may be located on substantially anyportion of frame 210. Sensor 240 may include a position sensor, aninertial measurement unit (IMU), a depth camera assembly, or anycombination thereof. In some embodiments, AR system 200 may or may notinclude sensor 240 or may include more than one sensor. In embodimentsin which sensor 240 includes an IMU, the IMU may generate calibrationdata based on measurement signals from sensor 240. Examples of sensor240 may include, without limitation, accelerometers, gyroscopes,magnetometers, other suitable types of sensors that detect motion,sensors used for error correction of the IMU, or some combinationthereof.

AR system 200 may also include a microphone array with a plurality ofacoustic sensors 220(A)-220(J), referred to collectively as acousticsensors 220. Acoustic sensors 220 may be transducers that detect airpressure variations induced by sound waves. Each acoustic sensor 220 maybe configured to detect sound and convert the detected sound into anelectronic format (e.g., an analog or digital format). The microphonearray in FIG. 2 may include, for example, ten acoustic sensors: 220(A)and 220(B), which may be designed to be placed inside a correspondingear of the user, acoustic sensors 220(C), 220(D), 220(E), 220(F),220(G), and 220(H), which may be positioned at various locations onframe 210, and/or acoustic sensors 220(I) and 220(J), which may bepositioned on a corresponding neckband 205.

The configuration of acoustic sensors 220 of the microphone array mayvary. While AR system 200 is shown in FIG. 2 as having ten acousticsensors 220, the number of acoustic sensors 220 may be greater or lessthan ten. In some embodiments, using higher numbers of acoustic sensors220 may increase the amount of audio information collected and/or thesensitivity and accuracy of the audio information. In contrast, using alower number of acoustic sensors 220 may decrease the computing powerrequired by the controller 250 to process the collected audioinformation. In addition, the position of each acoustic sensor 220 ofthe microphone array may vary. For example, the position of an acousticsensor 220 may include a defined position on the user, a definedcoordinate on the frame 210, an orientation associated with eachacoustic sensor, or some combination thereof.

Acoustic sensors 220(A) and 220(B) may be positioned on different partsof the user's ear, such as behind the pinna or within the auricle orfossa. Or, there may be additional acoustic sensors on or surroundingthe ear in addition to acoustic sensors 220 inside the ear canal. Havingan acoustic sensor positioned next to an ear canal of a user may enablethe microphone array to collect information on how sounds arrive at theear canal. By positioning at least two of acoustic sensors 220 on eitherside of a user's head (e.g., as binaural microphones), AR system 200 maysimulate binaural hearing and capture a 3D stereo sound field aroundabout a user's head. In some embodiments, acoustic sensors 220(A) and220(B) may be connected to AR system 200 via a wired connection, and inother embodiments, the acoustic sensors 220(A) and 220(B) may beconnected to AR system 200 via a wireless connection (e.g., a Bluetoothconnection). In still other embodiments, acoustic sensors 220(A) and220(B) may not be used at all in conjunction with AR system 200.

Acoustic sensors 220 on frame 210 may be positioned along the length ofthe temples, across the bridge, above or below display devices 215(A)and 215(B), or some combination thereof. Acoustic sensors 220 may beoriented such that the microphone array is able to detect sounds in awide range of directions surrounding the user wearing the AR system 200.In some embodiments, an optimization process may be performed duringmanufacturing of AR system 200 to determine relative positioning of eachacoustic sensor 220 in the microphone array.

AR system 200 may further include or be connected to an external device(e.g., a paired device), such as neckband 205. As shown, neckband 205may be coupled to eyewear device 202 via one or more connectors 230.Connectors 230 may be wired or wireless connectors and may includeelectrical and/or non-electrical (e.g., structural) components. In somecases, eyewear device 202 and neckband 205 may operate independentlywithout any wired or wireless connection between them.

While FIG. 2 illustrates the components of eyewear device 202 andneckband 205 in example locations on eyewear device 202 and neckband205, the components may be located elsewhere and/or distributeddifferently on eyewear device 202 and/or neckband 205. In someembodiments, the components of eyewear device 202 and neckband 205 maybe located on one or more additional peripheral devices paired witheyewear device 202, neckband 205, or some combination thereof.Furthermore, neckband 205 generally represents any type or form ofpaired device. Thus, the following discussion of neckband 205 may alsoapply to various other paired devices, such as smart watches, smartphones, wrist bands, other wearable devices, hand-held controllers,tablet computers, laptop computers, etc.

Pairing external devices, such as neckband 205, with AR eyewear devicesmay enable the eyewear devices to achieve the form factor of a pair ofglasses while still providing sufficient battery and computation powerfor expanded capabilities. Some or all of the battery power,computational resources, and/or additional features of AR system 200 maybe provided by a paired device or shared between a paired device and aneyewear device, thus reducing the weight, heat profile, and form factorof the eyewear device overall while still retaining desiredfunctionality.

For example, neckband 205 may allow components that would otherwise beincluded on an eyewear device to be included in neckband 205 since usersmay tolerate a heavier weight load on their shoulders than they wouldtolerate on their heads. Neckband 205 may also have a larger surfacearea over which to diffuse and disperse heat to the ambient environment.Thus, neckband 205 may allow for greater battery and computationcapacity than might otherwise have been possible on a stand-aloneeyewear device. Since weight carried in neckband 205 may be lessinvasive to a user than weight carried in eyewear device 202, a user maytolerate wearing a lighter eyewear device and carrying or wearing thepaired device for greater lengths of time than a user would toleratewearing a heavy standalone eyewear device, thereby enabling anartificial reality environment to be incorporated more fully into auser's day-to-day activities.

Neckband 205 may be communicatively coupled with eyewear device 202and/or to other devices. The other devices may provide certain functions(e.g., tracking, localizing, depth mapping, processing, storage, etc.)to AR system 200. In the embodiment of FIG. 2, neckband 205 may includetwo acoustic sensors (e.g., 220(I) and 220(J)) that are part of themicrophone array (or potentially form their own microphone subarray).Neckband 205 may also include a controller 225 and a power source 235.

Acoustic sensors 220(I) and 220(J) of neckband 205 may be configured todetect sound and convert the detected sound into an electronic format(analog or digital). In the embodiment of FIG. 2, acoustic sensors220(I) and 220(J) may be positioned on neckband 205, thereby increasingthe distance between the neckband acoustic sensors 220(I) and 220(J) andother acoustic sensors 220 positioned on eyewear device 202.

In some cases, increasing the distance between acoustic sensors 220 ofthe microphone array may improve the accuracy of beamforming performedvia the microphone array. For example, if a sound is detected byacoustic sensors 220(C) and 220(D) and the distance between acousticsensors 220(C) and 220(D) is greater than, e.g., the distance betweenacoustic sensors 220(D) and 220(E), the determined source location ofthe detected sound may be more accurate than if the sound had beendetected by acoustic sensors 220(D) and 220(E).

Controller 225 of neckband 205 may process information generated by thesensors on neckband 205 and/or AR system 200. For example, controller225 may process information from the microphone array that describessounds detected by the microphone array. For each detected sound,controller 225 may perform a DoA estimation to estimate a direction fromwhich the detected sound arrived at the microphone array.

As the microphone array detects sounds, controller 225 may populate anaudio data set with the information. In embodiments in which AR system200 includes an inertial measurement unit, controller 225 may computeall inertial and spatial calculations from the IMU located on eyeweardevice 202. Connector 230 may convey information between AR system 200and neckband 205 and between AR system 200 and controller 225. Theinformation may be in the form of optical data, electrical data,wireless data, or any other transmittable data form. Moving theprocessing of information generated by AR system 200 to neckband 205 mayreduce weight and heat in eyewear device 202, making it more comfortableto the user.

Power source 235 in neckband 205 may provide power to eyewear device 202and/or to neckband 205. Power source 235 may include, withoutlimitation, lithium ion batteries, lithium-polymer batteries, primarylithium batteries, alkaline batteries, or any other form of powerstorage. In some cases, power source 235 may be a wired power source.Including power source 235 on neckband 205 instead of on eyewear device202 may help better distribute the weight and heat generated by powersource 235.

As noted, some artificial reality systems may, instead of blending anartificial reality with actual reality, substantially replace one ormore of a user's sensory perceptions of the real world with a virtualexperience. One example of this type of system is a head-worn displaysystem, such as VR system 300 in FIG. 3, that mostly or completelycovers a user's field of view. VR system 300 may include a front rigidbody 302 and a band 304 shaped to fit around a user's head. VR system300 may also include output audio transducers 306(A) and 306(B).Furthermore, while not shown in FIG. 3, front rigid body 302 may includeone or more electronic elements, including one or more electronicdisplays, one or more inertial measurement units (IMUS), one or moretracking emitters or detectors, and/or any other suitable device orsystem for creating an artificial reality experience.

Artificial reality systems may include a variety of types of visualfeedback mechanisms. For example, display devices in AR system 200and/or VR system 300 may include one or more liquid crystal displays(LCDs), light emitting diode (LED) displays, organic LED (OLED)displays, and/or any other suitable type of display screen. Artificialreality systems may include a single display screen for both eyes or mayprovide a display screen for each eye, which may allow for additionalflexibility for varifocal adjustments or for correcting a user'srefractive error. Some artificial reality systems may also includeoptical subsystems having one or more lenses (e.g., conventional concaveor convex lenses, Fresnel lenses, adjustable liquid lenses, etc.)through which a user may view a display screen.

In addition to or instead of using display screens, some artificialreality systems may include one or more projection systems. For example,display devices in AR system 200 and/or VR system 300 may includemicro-LED projectors that project light (using, e.g., a waveguide) intodisplay devices, such as clear combiner lenses that allow ambient lightto pass through. The display devices may refract the projected lighttoward a user's pupil and may enable a user to simultaneously view bothartificial reality content and the real world. Artificial realitysystems may also be configured with any other suitable type or form ofimage projection system.

Artificial reality systems may also include various types of computervision components and subsystems. For example, AR system 100, AR system200, and/or VR system 300 may include one or more optical sensors suchas two-dimensional (2D) or three-dimensional (3D) cameras,time-of-flight depth sensors, single-beam or sweeping laserrangefinders, 3D LiDAR sensors, and/or any other suitable type or formof optical sensor. An artificial reality system may process data fromone or more of these sensors to identify a location of a user, to mapthe real world, to provide a user with context about real-worldsurroundings, and/or to perform a variety of other functions.

Artificial reality systems may also include one or more input and/oroutput audio transducers. In the examples shown in FIGS. 1 and 3, outputaudio transducers 108(A), 108(B), 306(A), and 306(B) may include voicecoil speakers, ribbon speakers, electrostatic speakers, piezoelectricspeakers, bone conduction transducers, cartilage conduction transducers,and/or any other suitable type or form of audio transducer. Similarly,input audio transducers 110 may include condenser microphones, dynamicmicrophones, ribbon microphones, and/or any other type or form of inputtransducer. In some embodiments, a single transducer may be used forboth audio input and audio output.

While not shown in FIGS. 1-3, artificial reality systems may includetactile (i.e., haptic) feedback systems, which may be incorporated intoheadwear, gloves, body suits, handheld controllers, environmentaldevices (e.g., chairs, floormats, etc.), and/or any other type of deviceor system. Haptic feedback systems may provide various types ofcutaneous feedback, including vibration, force, traction, texture,and/or temperature. Haptic feedback systems may also provide varioustypes of kinesthetic feedback, such as motion and compliance. Hapticfeedback may be implemented using motors, piezoelectric actuators,fluidic systems, and/or a variety of other types of feedback mechanisms.Haptic feedback systems may be implemented independent of otherartificial reality devices, within other artificial reality devices,and/or in conjunction with other artificial reality devices.

By providing haptic sensations, audible content, and/or visual content,artificial reality systems may create an entire virtual experience orenhance a user's real-world experience in a variety of contexts andenvironments. For instance, artificial reality systems may assist orextend a user's perception, memory, or cognition within a particularenvironment. Some systems may enhance a user's interactions with otherpeople in the real world or may enable more immersive interactions withother people in a virtual world.

Artificial reality systems may also be used for educational purposes(e.g., for teaching or training in schools, hospitals, governmentorganizations, military organizations, business enterprises, etc.),entertainment purposes (e.g., for playing video games, listening tomusic, watching video content, etc.), and/or for accessibility purposes(e.g., as hearing aids, visuals aids, etc.). The embodiments disclosedherein may enable or enhance a user's artificial reality experience inone or more of these contexts and environments and/or in other contextsand environments.

Some artificial reality systems may map a user's and/or device'senvironment using techniques referred to as “simultaneous location andmapping” (SLAM). SLAM mapping and location identifying techniques mayinvolve a variety of hardware and software tools that can create orupdate a map of an environment while simultaneously keeping track of auser's location within the mapped environment. SLAM may use manydifferent types of sensors to create a map and determine a user'sposition within the map.

SLAM techniques may, for example, implement optical sensors to determinea user's location. Radios including WiFi, Bluetooth, global positioningsystem (GPS), cellular or other communication devices may be also usedto determine a user's location relative to a radio transceiver or groupof transceivers (e.g., a WiFi router or group of GPS satellites).Acoustic sensors such as microphone arrays or 2D or 3D sonar sensors mayalso be used to determine a user's location within an environment.

AR and VR devices (such as systems 100, 200, and 300 of FIGS. 1-3,respectively) may incorporate any or all of these types of sensors toperform SLAM operations such as creating and continually updating mapsof the user's current environment. In at least some of the embodimentsdescribed herein, SLAM data generated by these sensors may be referredto as “environmental data” and may indicate a user's currentenvironment. This data may be stored in a local or remote data store(e.g., a cloud data store) and may be provided to a user's AR/VR deviceon demand.

The following will provide, with reference to FIG. 4, detaileddescriptions of computer-implemented methods for (1) performing aninterest segmentation and (2) generating virtual content based on theresults of the interest segmentation. Detailed descriptions ofcorresponding example systems (e.g., AR system 500) will also beprovided in connection with FIG. 5. In addition, detailed descriptionsof exemplary embodiments of the disclosed systems and methods will beprovided in connection with FIG. 6.

FIG. 4 is a flow diagram of an exemplary computer-implemented method400. The steps shown in FIG. 4 may be performed by any suitablecomputer-executable code and/or computing system, such as the systemsdescribed herein. In one embodiment, the steps shown in FIG. 4 may beperformed by modules operating within a computing device capable ofreading computer-executable instructions, such as computing device 502and/or AR device 504 in FIG. 5. In some embodiments, each of the modulesmay operate within a single device. In other embodiments, the modulesmay be spread out across multiple devices.

Computing device 502 may represent any type or form of device capable ofreading computer-executable instructions. In some examples, computingdevice 502 may represent a backend computing device. For example,computing device 502 may represent a server maintained by an AR platformthat performs one or more AR functions (including, for example,collecting user data such as eye-tracking data, creating personalizedprofiles for users, and/or generating and/or transmitting virtualcontent to AR devices based on the personalized profiles). In anotherexample, computing device 502 may represent a client device (e.g., usedto receive user input indicating one or more preferences of the userand/or user input granting permission for certain operations).

AR device 504 may represent any type or form of device, capable ofreading computer-executable instructions, that performs one or moreartificial reality functions. In one example, AR device 504 mayrepresent a wearable device with a display element (e.g., a head-mounteddisplay) that presents virtual stimuli (e.g., virtual visual stimuli) toa user. Exemplary embodiments of AR device 504 include, withoutlimitation, AR system 100 in FIG. 1, AR system 200 in FIG. 2, and VRsystem 300 in FIG. 3.

As illustrated in FIG. 4, at step 410, one or more of the systemsdescribed herein may perform a semantic segmentation of an image of auser's environment, captured by an AR device being worn by the user, toidentify objects within the user's environment. For example, asillustrated in FIG. 5, AR device 504 may be worn by a user 506 and maycapture an image 508 of user 506's real-world environment (e.g., animage of a portion of the environment within user 506's field of view).Then, a segmentation module 510 may perform a semantic segmentation ofimage 508 to identify and label objects 512 within the environment.

The term “semantic segmentation” may refer to any type or form ofdigital process for identifying which objects are captured in an image.A semantic segmentation process may include partitioning a digital imageby associating each pixel within the digital image with a class label(e.g., tree, child, sandwich, guitar, etc.). As a specific example, atthe time of step 410, user 506 may be located in a real-worldenvironment 600 (depicted in FIG. 6) and AR device 504 may capture animage of environment 600. Then, segmentation module 510 may analyze theimage to identify the various objects included within the image (e.g.,coffee shop 602, sidewalk 604, man 606 next to bicycle 608, chimney 610,cloud 612, road 614, etc.).

Segmentation module 510 may identify and label objects 512 in a varietyof ways. In some examples, segmentation module 510 may identify andlabel objects 512 using deep learning. In one such example, segmentationmodule 510 may include an encoder network and a decoder network. Theencoder network may represent a pre-trained classification network. Thedecoder network may semantically project the features learned by theencoder network to the pixel space of image 508 to classify objects 512.The decoder network may utilize a variety of approaches to classifyobjects 512 (e.g., a region-based approach, a fully convolutionalnetwork (FCN) approach, etc.).

In addition to performing a semantic segmentation of the image of theuser's environment, one or more of the systems described herein mayperform an interest segmentation of the image to determine a personalinterest that the user may have in a particular object identified viathe semantic segmentation (step 420 in FIG. 4). For example, asillustrated in FIG. 5, segmentation module 510 may perform an interestsegmentation of image 508 to determine a personal interest that user 506may have in one or more of objects 512. In this example, segmentationmodule 510 may determine (e.g., predict) that user 506 may have apersonal interest in a particular object 513. The term “interestsegmentation” may refer to any type or form of digital process foridentifying a personal interest that a user may have in an objectidentified via a semantic segmentation process.

Segmentation module 510 may determine that user 506 has a variety ofdifferent kinds of personal interest in particular object 513. In oneexample, segmentation module 510 may determine that particular object513 is a type of object in which user 506 is interested. As a specificexample, user 506 may be interested in guitars and particular object 513may represent a guitar.

In another example, segmentation module 510 may determine thatparticular object 513 belongs to user 506 (e.g., a set of keys thatbelong to user 506). Additionally or alternatively, segmentation module510 may determine that user 506 has interacted with particular object513 more than a threshold amount (e.g., more than a threshold number oftimes, a threshold cumulative amount of time, and/or a thresholdfrequency). As a specific example, particular object 513 may represent aparticular tree that user 506 always stops to look at when he walksthrough a particular park. In this specific example, the personalinterest identified and/or predicted by segmentation module 510 may bethat particular object 513 always attracts the gaze of user 506.

As another example, segmentation module 510 may determine thatparticular object 513 relates to another person, place, or thing thatinterests user 506. As a specific example, user 506 may have an interestin 20th-century American literature and particular object 513 mayrepresent a home that was once inhabited by a famous 20^(th)-centuryAmerican author.

Segmentation module 510 may predict user 506's personal interest inparticular object 513 (that is, may perform the interest segmentation)based on one or more of a variety of factors, or on any relevantcombination of factors. In one example, these factors may include user506's historical eye-tracking data (e.g., collected by AR device 504while being worn by user 506). In this example, segmentation module 510may operate based on a setting and/or policy that objects that haveattracted a user's gaze (e.g., eye-gaze) in the past are predictive ofwhat will interest the user in the future. Additionally or alternativelyin this example, segmentation module 510 may operate based on a settingand/or policy that a particular object that has historically attracted auser's gaze (e.g., eye-gaze) more than a threshold amount belongs to theuser and/or has a particular significance to the user.

As a specific example, AR device 504 may have collected eye-trackingdata indicating that user 506 has visually focused on a particular typeof object (e.g., butterflies) more than a threshold amount. In thisspecific example, particular object 513 may represent the particulartype of object (e.g., a butterfly) and segmentation module 510 maydetermine that user 506 may be interested in particular object 513 basedon user 506 having visually focused on the particular type of object(e.g., butterflies) more than a threshold amount in the past.

As another specific example, AR device 504 may have collectedeye-tracking data indicating that user 506 has visually focused onparticular object 513 itself in the past more than a threshold amount.In this specific example, segmentation module 510 may determine thatparticular object 513 belongs to user 506, and/or that particular object512 has a particular significance to user 506, based on user 506 havingvisually focused on particular object 513 in the past more than thethreshold amount.

In a related embodiment, the factors influencing the interestsegmentation may include eye-tracking data indicating a current eye-gazeof user 506 (e.g., which may be shown in heat map generated based on theuser's eye-tracking data). In this embodiment, user 506's current eyegaze may be focused on particular object 513. Additionally oralternatively, user 506's current eye gaze may express a gaze patternthat indicates that user 506 may be interested in particular object 513.FIG. 6 may be used as a specific example of a gaze pattern thatindicates a potential interest in particular object 513. In thisspecific example, AR device 504 may collect eye-tracking data indicatingthat user 506 is visually scanning along the store front abuttingsidewalk 604 (e.g., after having recently used a GPS application tosearch for “Café Quotidien” and/or at a time during which user 506normally has a coffee). Then, based at least in part on thiseye-tracking data, segmentation module 510 may determine that user 506may have an interest in coffee shop 602.

In some examples, the user's historical or current eye-tracking data maybe collected by an eye-tracking module 514 of AR device 504.Eye-tracking module 514 may identify user 506's eye-tracking using anytype or form of eye-tracking technology. For example, eye-trackingmodule 514 may rely on a device embedded in and/or functioning inconnection with AR device 504 to transmit light from a light source(such as infrared light from an infrared emitter) at the eyes of user506. In this example, eye-tracking module 514 may rely on one or moresensors embedded within AR device 504 to identify a reflection of thelight source from the eyes. Then, eye-tracking module 514 may analyzethe reflection to determine the direction of the user's eye-gaze. In oneexample, eye-tracking module 514 may identify pixel coordinates, on adisplay element of AR device 504, at which user 506 is gazing. Then,eye-tracking module 514 may detect an object corresponding to the pixelcoordinates (e.g., an object that was identified and labeled via asemantic segmentation process) and determine that user 506 is gazing atthe detected object.

In some embodiments, the factors influencing the interest segmentationmay include data describing an interaction that user 506 has had with anobject. In some examples, the data describing the interaction may becollected by AR device 504. For example, AR device 504 may recordeye-tracking data (e.g., using eye-tracking module 514) and/or hapticdata (e.g., using a haptic device that operates as part of or inconnection with AR device 504) revealing user 506 handling one or moreobjects (e.g., a certain phone). In additional or alternative examples,the data describing the interaction may be recorded by a device thatoperates in connection with AR device 504 (e.g., within an Internet ofThings (IoT) system). As a specific example, a smart toaster within asame IoT system as AR device 504 may record that user 506 toasts a bagelevery morning at around 6 am.

In some examples, the factors influencing the interest segmentation mayinclude data (e.g., historical eye-tracking data) of one or moreadditional users (e.g., users with a particular similarity or bundle ofsimilarities to user 506). As a specific example, user 506 may identifyas a guitar enthusiast and segmentation module 510 may at leastpartially base its interest determination for user 506 on theeye-tracking data of other guitar enthusiasts (e.g., an object visuallylooked at or physically handled by other guitar enthusiasts may bepredicted to be of interest to user 506). In some examples, theadditional users may be selected (i.e., users predicted to have asimilarity to user 506 may be selected) based on a personal profile ofuser 506 using machine learning.

Additional factors that may influence the interest segmentation mayinclude, without limitation, GPS data associated with user 506, URLbrowsing data associated with user 506, a digital purchase history ofuser 506, a travel history of user 506, digital content generated byuser 506 (e.g., email content, calendar content, social media content,etc.), and/or user preference data submitted by user 506 (e.g.,submitted as part of a registration process and/or as part of setting upa personal profile).

In certain embodiments, one or more ephemeral factors (e.g., factorspredicted to be affecting a current state of user 506) may influence theinterest segmentation. Such ephemeral factors may include, withoutlimitation, a time of day, a recent activity of the user, and/or acurrent activity of user 506. Using FIG. 6 as a specific example, theinterest segmentation may predict that user 506 is likely to have aninterest in coffee shop 602 in the morning as he is heading to work butis unlikely to have an interest in coffee shop 602 in the evening aftereating dinner.

In one embodiment, an ephemeral factor may include ephemeralphysiological data (e.g., heart-rate and/or brain wave data) collectedby a sensor that monitors user 506's physiologic state (e.g., a sensorembedded within AR device 504). In some examples, the physiological datamay be used to deduce an emotional state of user 506, which may be usedas part of the interest segmentation. In additional or alternativeexamples, an emotional state may be identified via user input.

Segmentation module 510 may use the factors described above to performthe interest segmentation in a variety of ways. In some examples,segmentation module 510 may perform the interest segmentation using aneural network. In these examples, one or more of the factors describedabove may serve as inputs to the neural network. In other examples,segmentation module 510 may perform the interest segmentation using oneor more rigid rules in a policy. A specific example of a rigid rule mayinclude a rule to flag, as a personally significant object, any objectthat has attracted a user's gaze more than a predetermined amount in thepast.

In one example, a personal profile module 516 may have created apersonal profile 518 for user 506. Personal profile 518 may be generatedbased on a variety of factors, such as the factors described above. Thispersonal profile may then be used to perform the interest segmentation.In some embodiments, personal profile 518 may be generated and/ormaintained by a backend server. In other embodiments, personal profile518 may be generated and/or maintained by AR device 504.

The factors described above are intended to be illustrative of exemplarytypes of factors that may be used as part of an interest segmentation.Segmentation module 510 may use any of the described factors, either inisolation or in combination, to perform an interest segmentation. Insome examples, segmentation module 510 may rely on a combination offactors that includes one or more of the factors described abovetogether with one or more additional factors not explicitly describedherein.

Returning to FIG. 1, at step 430, one or more of the systems describedherein may create virtual content relating to the particular objectbased on the user's personal interest in the particular object. Forexample, as illustrated in FIG. 5, a content creation module 520 maycreate virtual content 522 relating to particular object 513.

Virtual content 522 may represent any type of computer-generated contentconfigured to be visually displayed via a display element 526 of ARdevice 504. Virtual content 522 may take a variety of different forms(e.g., a digital graphic, a filter, and/or text) and may communicate avariety of different information, depending on the kind of interest user506 is predicted to have in particular object 513.

In some examples, virtual content 522 may include content configured todraw user 506's attention to particular object 513. Using FIG. 6 as aspecific example, particular object 513 may represent a coffee house(i.e., coffee shop 602) and segmentation module 510's interestsegmentation may have predicted that user 506 may be interested incoffee shop 602 (e.g., based on data indicating that user 506 usuallydrinks coffee at this time, eye-tracking data indicative of user 506scanning his environment looking for something, and/or data indicatingthat users who match a profile of user 506 rate coffee shop 602 highly).In this specific example, content creation module 520 may be configuredto create content (e.g., a digital graphic of a cup of coffee) thatdraws user 506's attention to coffee shop 602 based on the results ofthe interest segmentation.

In one embodiment, virtual content 522 may include content that providesinformation about particular object 513 (e.g., historical information,hours of operation, consumer ratings, how-to information, interestingfacts, etc.). As a specific example, particular object 513 may representa particular tree in a park and eye-tracking module 514 may havecollected eye-tracking data indicating that user 206 has gazed at theparticular tree more than a threshold amount. In this example, contentcreation module 520 may be configured to create content describing acommon and/or scientific name of the tree, based on an interestsegmentation that suggests user 506's interest in the tree based on user506's historical eye-tracking data.

As another specific example, particular object 513 may represent thepump filter of a washing machine and eye-tracking module 514 may havecollected eye-tracking data indicating that user 506 has opened the backof the washing machine and is handling the pump filter. In this example,content creation module 520 may be configured to create content relatingto how to clean and/or replace a pump filter, based on an interestsegmentation that suggests that user 506 may be attempting to cleanand/or replace the pump filter.

In some examples, virtual content 522 may include content that providesinformation about a location of particular object 513. In theseexamples, virtual content 522 may indicate a location of particularobject 513 within the environment and may be created in response to adetermination that user 506 has lost particular object 513. For example,segmentation module 510 may have determined a location of particularobject 513 at a first moment in time during which particular object 513is within user 506's field of view, while user 506 was wearing AR device504 (e.g., while segmentation module 510 was performing the semanticsegmentation and the interest segmentation). Then, at a second moment intime, user 506 may be determined to have lost particular object 513. Inresponse to this determination, content creation module 520 may createvirtual content that indicates the location of particular object 513 atthe first moment in time. In this example, virtual content 522 mayinclude text describing the location (e.g., the text “your keys may beon your kitchen counter”) or a visual image (captured at the firstmoment in time) showing the last known location of particular object 513(e.g., an image of keys on a kitchen counter).

In one example, virtual content 522 may include a reminder. As aspecific example, segmentation module 510 may have identified a paper ona kitchen counter via the semantic segmentation in step 410. Then, (atstep 420) segmentation module 510 may have determined that the paper isa bill that is past due. (This determination may be based on datacollected by AR device 504 at a previous time during which user 506 hadthe contents of the paper within AR device 504's field of view). In thisexample, virtual content 522 may include a reminder to pay the bill.

Content creation module 520 may create virtual content 522 in responseto a variety of events. In some embodiments, content creation module 520may automatically create virtual content 522 in response to apolicy-driven trigger, without any prompting from user 506 for the same.Such a trigger may include, without limitation, a prediction that anobject within user 506's field of view may be of interest to the userand/or a prediction or determination that user 506 has lost an object.In other embodiments, virtual content 522 may be created in response toreceiving user input requesting the same (e.g., a user request forinformation relating to the location of a lost object, a user request toidentify a nearby coffee house, a user request to receive a virtualreminder, etc.).

In some examples, content creation module 520 may rely on machinelearning to identify content that is likely to be of interest to theuser (e.g., based on the factors described above in connection with step420). In one such example, a neural network may (1) identify a personalsignificance that an object in user 506's environment may have to user506 (as part of the interest segmentation in step 420) and then (2)identify content (related to the object) that may be of interest to user506 (as part of the content creation in step 430). As a specificexample, a neural network may (1) determine that a sycamore tree in user506's environment is likely to be of interest (e.g., based on user 506'shistorical interest in the tree) and (2) determine that a poem writtenabout sycamore trees may be of interest to user 506 (e.g., based on adetermination that user 506 is interested in poetry).

Returning to FIG. 4, after creating the virtual content, one or more ofthe systems described herein may display the virtual content within adisplay element of the AR device. For example, as illustrated in FIG. 5,display module 524 may display virtual content 522 via display element526 of AR device 504.

Display module 524 may display virtual content 522 in a variety of ways.In some examples, display module 524 may display virtual content 522 ina manner that visually associates virtual content 522 with particularobject 513. For example, display module 524 may display virtual content522 such that virtual content 522 is superimposed over particular object513 within display element 526 and/or hovers substantially above, below,or to the side of particular object 513 within a predetermined distanceof particular object 513. In this example, display module 524 maydisplay virtual content 522 at pixel coordinates within a screen ofdisplay element 526 that correspond to particular object 513 within user506's environment (e.g., that correspond to an area that superimposesparticular object 513 or hovers above, below or to the side ofparticular object 513).

Display module 524 may display virtual content 522 via a variety ofphysical elements. For example, display module 524 may display virtualcontent 522 using one or more components of AR device 504 in FIG. 5, ARsystem 100 in FIG. 1, AR system 200 in FIG. 2, and/or VR system 300 inFIG. 3. Using FIG. 2 as a specific example, display module 524 maydisplay virtual content 522 via left display device 215(A) and/or rightdisplay device 215(B) in AR system 200.

Each component (e.g., each step and/or sub-step) of method 400 may beperformed in response to receiving user permission to do so (e.g., wherea component is not performed in the absence of such permission). In oneexample, the disclosed systems and methods may include a permissionsprocess for obtaining the user's permission. During this process, adescription of each component may be digitally presented to a user(e.g., user 506 in FIG. 5) and the user may select which of thecomponents to permit. In one embodiment, components may be bundledtogether and presented to the user as a bundle and/or a series ofbundles, allowing the user to grant or deny permission to any givenbundle. In another embodiment, each component may be individuallypresented to the user, allowing the user to grant or deny permission toeach individual component. In the absence of a selection to grantpermission (that is to opt-in) to a particular component or bundle ofcomponents, a default permission-denial may be selected on behalf of theuser.

In one embodiment, the permissions process may be performed ex ante(e.g., permission to perform an operation may be digitally solicited andreceived from the user prior to performing the operation). In thisembodiment, the user's permission may be general (e.g., the user may begiven the option to allow a type of operation to be performed). Specificexamples of a general permission may include, without limitation,permission to create a personalized profile, permission to collecteye-tracking data, permission to add the user's data to an aggregatedatabase, permission to create and display personalized virtual contentto the user, etc.

Additionally or alternatively, the permissions process may be performedpost hoc. In this embodiment, the user's permission may be specific(e.g., the user may be given the option to allow a specific piece ofdata to be collected). As a specific example of a post-hoc permissionsprocess, the user may be digitally transmitted a periodic summary ofdata collected relating to him or her (e.g., a daily summary, a weeklysummary, a monthly summary, etc.). Specific examples of such data mayinclude eye-tracking data, browsing history, and/or GPS data collectedduring the period (e.g., collected by the user's AR device and/or by anadditional device such as a smartphone or a laptop).

The user may grant permission for the summarized data to be used in avariety of different ways. For example, the user may grant permission toallow the data to (1) be transmitted from the user's AR device and/oradditional device to a backend server, (2) be used to perform anoperation (such as creating a personalized profile), and/or (3) beanonymized and added to an aggregate databased including the anonymizeddata of many different users. In some examples, the user may customizewhich uses he or she permits, allowing some and denying others ifdesired.

As described throughout the present disclosure, the disclosed systemsand methods may provide systems and methods for delivering personalizedvirtual content to a user of an AR device, such as a pair of AR glasses,who has opted-in to receiving personalized information. In someexamples, the personalized content may relate to a particular objectwithin the user's environment predicted or known to be of interest tothe user (e.g., based on the particular object being of a particulartype and/or based on the particular object belonging to the user and/orhaving a personal significance to the user). In these examples, thedisclosed systems and methods may perform a semantic segmentation of theuser's environment to determine which objects are included in the user'senvironment and an interest segmentation to determine a significancethat one of more of the objects may have to the user. Then, thedisclosed systems and methods may generate and display content createdbased on the results of the interest segmentation.

In some examples, the personalized content may be passively presented tothe user. That is, the content may be presented without explicit userrequest for the same. As an example, the AR device may present thecontent in response to detecting certain objects in the user'senvironment. In other examples, the personalized content may bepresented to the user in response to receiving a user query for thesame.

The disclosed systems and methods may identify objects of personalsignificance and select content to display to the user based on avariety of inputs. For example, in some examples, the user may haveopted-in to the AR device collecting eye-gazing and/or hand-trackingdata. In these examples, the objects may be identified, and/or thecontent may be selected, based on the eye-gazing and/or thehand-tracking data collected by the AR device.

In some embodiments, the user may have opted-in to receivingcontext-driven predicted content. In these embodiments, the identifiedobjects and/or the selected content may vary based on contextualfactors, such as a time of day and/or an activity the user is engagedin. In one embodiment, the user may have opted-in to allowing the ARdevice to deduce the user's emotional state (e.g., based on physiologicdata collected by the AR device). In this example, predictions may bebased on the user's current emotional state.

In some embodiments, a personalized profile may be created for the userbased on the inputs that the user has opted-in to providing. In one suchembodiment, anonymized data of other users may be collected via crowdsurfing and used to predict objects and/or content that may be ofinterest to the user. For example, a user may be categorized into agroup based on one or more inputs to the user's personalized profilesand the data of other users within the group may be used to predict whatcontent will be of interest to the user.

As detailed above, the computing devices and systems described and/orillustrated herein broadly represent any type or form of computingdevice or system capable of executing computer-readable instructions,such as those contained within the modules described herein. In theirmost basic configuration, these computing device(s) may each include atleast one memory device and at least one physical processor.

In some examples, the term “memory device” may refer to any type or formof volatile or non-volatile storage device or medium capable of storingdata and/or computer-readable instructions. In one example, a memorydevice may store, load, and/or maintain one or more of the modulesdescribed herein. Examples of memory devices include, withoutlimitation, Random Access Memory (RAM), Read Only Memory (ROM), flashmemory, Hard Disk Drives (HDDs), Solid-State Drives (SSDs), optical diskdrives, caches, variations or combinations of one or more of the same,or any other suitable storage memory.

In addition, the term “physical processor” may, in some examples, referto any type or form of hardware-implemented processing unit capable ofinterpreting and/or executing computer-readable instructions. In oneexample, a physical processor may access and/or modify one or moremodules stored in the above-described memory device. Examples ofphysical processors include, without limitation, microprocessors,microcontrollers, Central Processing Units (CPUs), Field-ProgrammableGate Arrays (FPGAs) that implement softcore processors,Application-Specific Integrated Circuits (ASICs), portions of one ormore of the same, variations or combinations of one or more of the same,or any other suitable physical processor.

Although illustrated as separate elements, the modules described and/orillustrated herein may represent portions of a single module orapplication. In addition, in certain embodiments one or more of thesemodules may represent one or more software applications or programsthat, when executed by a computing device, may cause the computingdevice to perform one or more tasks. For example, one or more of themodules described and/or illustrated herein may represent modules storedand configured to run on one or more of the computing devices or systemsdescribed and/or illustrated herein. One or more of these modules mayalso represent all or portions of one or more special-purpose computersconfigured to perform one or more tasks.

In addition, one or more of the modules described herein may transformdata, physical devices, and/or representations of physical devices fromone form to another. For example, one or more of the modules recitedherein may transform a processor, volatile memory, non-volatile memory,and/or any other portion of a physical computing device from one form toanother by executing on the computing device, storing data on thecomputing device, and/or otherwise interacting with the computingdevice.

The term “computer-readable medium” may, in some examples, refer to anyform of device, carrier, or medium capable of storing or carryingcomputer-readable instructions. Examples of computer-readable mediainclude, without limitation, transmission-type media, such as carrierwaves, and non-transitory-type media, such as magnetic-storage media(e.g., hard disk drives, tape drives, and floppy disks), optical-storagemedia (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), andBLU-RAY disks), electronic-storage media (e.g., solid-state drives andflash media), and other distribution systems.

The process parameters and sequence of the steps described and/orillustrated herein are given by way of example only and can be varied asdesired. For example, while the steps illustrated and/or describedherein may be shown or discussed in a particular order, these steps donot necessarily need to be performed in the order illustrated ordiscussed. The various exemplary methods described and/or illustratedherein may also omit one or more of the steps described or illustratedherein or include additional steps in addition to those disclosed.

The preceding description has been provided to enable others skilled inthe art to best utilize various aspects of the exemplary embodimentsdisclosed herein. This exemplary description is not intended to beexhaustive or to be limited to any precise form disclosed. Manymodifications and variations are possible without departing from thespirit and scope of the present disclosure. The embodiments disclosedherein should be considered in all respects illustrative and notrestrictive. Reference should be made to the appended claims and theirequivalents in determining the scope of the present disclosure.

Unless otherwise noted, the terms “connected to” and “coupled to” (andtheir derivatives), as used in the specification and claims, are to beconstrued as permitting both direct and indirect (i.e., via otherelements or components) connection. In addition, the terms “a” or “an,”as used in the specification and claims, are to be construed as meaning“at least one of.” Finally, for ease of use, the terms “including” and“having” (and their derivatives), as used in the specification andclaims, are interchangeable with and have the same meaning as the word“comprising.”

What is claimed is:
 1. A computer-implemented method comprising:performing a semantic segmentation of an image of a user's environment,captured by an artificial reality (AR) device being worn by the user, toidentify objects within the user's environment; in addition toperforming the semantic segmentation, performing an interestsegmentation of the image to identify a personal significance that aparticular object, identified via the semantic segmentation, may have tothe user; creating virtual content relating to the particular objectbased on the identified personal significance of the particular objectto the user; and displaying the virtual content within a display elementof the AR device.
 2. The computer-implemented method of claim 1, whereinperforming the interest segmentation comprises: identifying a type ofobject corresponding to the particular object; and determining that theuser is interested in the type of object.
 3. The computer-implementedmethod of claim 2, wherein the virtual content comprises contentconfigured to draw the user's attention to the particular object.
 4. Thecomputer-implemented method of claim 1, wherein identifying the personalsignificance comprises determining that at least one of: the particularobject belongs to the user; or the user has interacted with theparticular object more than a threshold amount.
 5. Thecomputer-implemented method of claim 4, wherein: the virtual content iscreated and displayed in response to determining that the user has lostthe particular object; and the virtual content is configured to indicatea location of the particular object within the environment.
 6. Thecomputer-implemented method of claim 5, wherein: the semantic andinterest segmentations are performed at a first moment in time, duringwhich the user wearing the AR device is within the environment whilewearing the AR device; and the user is determined to have lost theparticular object at a second moment in time during which the user is nolonger within the environment.
 7. The computer-implemented method ofclaim 1, wherein the interest segmentation is based on at least one of:the user's historical eye-tracking data; or historical eye-tracking dataof one or more additional users with a particular similarity or bundleof similarities to the user.
 8. The computer-implemented method of claim1, wherein the interest segmentation is based on a heatmap of theenvironment comprising the user's eye-tracking data with respect to theenvironment.
 9. The computer-implemented method of claim 1, wherein theinterest segmentation is based on at least one of: GPS data associatedwith the user; URL browsing data associated with the user; oruser-submitted data submitted by the user.
 10. The computer-implementedmethod of claim 1, wherein the interest segmentation is based on aninteraction of the user recorded by an additional device within anInternet of Things (IoT) system that comprises the AR device.
 11. Thecomputer-implemented method of claim 1, wherein the interestsegmentation is based on data describing ephemeral factors predicted tobe affecting a current state of the user.
 12. The computer-implementedmethod of claim 11, wherein the ephemeral factors comprise at least oneof: a time of day; an emotional state of the user; a physiological stateof the user; or a current activity of the user.
 13. Thecomputer-implemented method of claim 1, wherein the interestsegmentation is performed by a neural network.
 14. Thecomputer-implemented method of claim 1, wherein displaying the virtualcontent within the display element comprises displaying the virtualcontent such that the virtual content is superimposed over theparticular object within the display element.
 15. Thecomputer-implemented method of claim 1, wherein the interestsegmentation is performed and the virtual content is displayed inresponse to receiving permission from the user to perform the interestsegmentation and display the virtual content.
 16. A system comprising: asegmentation module, stored in memory, that: performs a semanticsegmentation of an image of a user's environment, captured by anartificial reality (AR) device being worn by the user, to identifyobjects within the user's environment; and in addition to performing thesemantic segmentation, performs an interest segmentation of the image toidentify a personal significance that a particular object, identifiedvia the semantic segmentation, may have to the user; a content creationmodule, stored in memory, that creates virtual content relating to theparticular object based on the identified personal significance of theparticular object to the user; and a display module, stored in memory,that displays the virtual content within a display element of the ARdevice; and at least one physical processor configured to execute thesegmentation module, the content creation module, and the displaymodule.
 17. The system of claim 16, wherein the segmentation moduleperforms the interest segmentation by: identifying a type of objectcorresponding to the particular object; and determining that the user isinterested in the type of object.
 18. The system of claim 17, whereinthe virtual content comprises content configured to draw the user'sattention to the particular object.
 19. The system of claim 16, whereinthe interest segmentation is based on at least one of: the user'shistorical eye-tracking data; or historical eye-tracking data of one ormore additional users with a particular similarity or bundle ofsimilarities to the user.
 20. A non-transitory computer-readable mediumcomprising one or more computer-readable instructions that, whenexecuted by at least one processor of a computing device, cause thecomputing device to: perform a semantic segmentation of an image of auser's environment, captured by an artificial reality (AR) device beingworn by the user, to identify objects within the user's environment; inaddition to performing the semantic segmentation, perform an interestsegmentation of the image to identify a personal significance that aparticular object, identified via the semantic segmentation, may have tothe user; create virtual content relating to the particular object basedon the identified personal significance of the particular object to theuser; and display the virtual content within a display element of the ARdevice.